{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.3 监控指标可视化\n",
    "在11.2节中着重介绍了通过TensorBoard的**GRAPHS可视化TensorFlow计算图的结构以及在计算图上的信息。**TensorBoard除了可以可视化TensorFlow的计算图，还可以可视化TensorFlow程序运行过程中各种有助于了解程序运行状态的监控指标。在本节中将介绍如何利用TensorBoard中其他栏目可视化这些监控指标。\n",
    "\n",
    "**除GRAPHS以外，TensorBoard界面中还提供SCALARS、IMAGES、AUDIO、DISTRIBUTIONS、HISTOGRAMS和TEXT六个界面来可视化其他的监控指标。**以下程序展示了如何将TensorFlow程序运行时的信息输出到TensorBoard日志文件中。因为需要在变量定义时加上日志输出，所以这里先不共用5.5节中的mnist_inference.py。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-1-7516fa21a43d>:63: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ../../datasets/MNIST_data\\train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ../../datasets/MNIST_data\\train-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.one_hot on tensors.\n",
      "Extracting ../../datasets/MNIST_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting ../../datasets/MNIST_data\\t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "SUMMARY_DIR = \"log/11.3.log\"\n",
    "BATCH_SIZE = 100\n",
    "TRAIN_STEPS = 3000\n",
    "\n",
    "# 1. 生成变量监控信息并定义生成监控信息日志的操作。\n",
    "# 其中var给出了需要记录的张量，name给出了在可视化结果中显示的图发名称，这个名称一般与变量名一致。\n",
    "def variable_summaries(var, name):\n",
    "    # 将生成监控信息的操作放到统一命名空间下\n",
    "    with tf.name_scope('summaries'):\n",
    "        # 通过tf.summary.histogram函数记录张量中元素的取值分布。对于给出的图表\n",
    "        # 名称和张量，tf.summary.histogram函数会生成一个Summary protocol buffer。\n",
    "        # 将Summary写入TensorBoard日志文件后，在HISTOGRAMS栏和DISTRIBUTION栏下\n",
    "        # 都会出现对应名称的图表。和TensorFlow中其他操作类似，\n",
    "        # tf.summary.histogram函数不会立刻被执行，只有当sess.run函数明确调用这\n",
    "        # 个操作时，TensorFlow才会具正生成并输出Summary protocol buffer。\n",
    "        # 下文将更加详细地介绍如何理解HISTOGRAMS栏和DISTRIBUTION栏下的信息。\n",
    "        tf.summary.histogram(name, var)\n",
    "        \n",
    "        # 计算变量的平均值，并定义生成平均值信息日志的操作。记录变量平均值信息的日志标签名\n",
    "        # 为'mean/'＋name，其中mean为命名空间，/是命名空间的分隔符，从图11.14中可以看到，\n",
    "        # 在相同命名空间中的监控指标会被整合到同一栏中；name则给出了当前监控指标属于哪一个变量。\n",
    "        mean = tf.reduce_mean(var)\n",
    "        tf.summary.scalar('mean/' + name, mean)\n",
    "        \n",
    "        # 计算变量的标准差，并定义生成其日志的操作\n",
    "        stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))\n",
    "        tf.summary.scalar('stddev/' + name, stddev)  \n",
    "        \n",
    "\n",
    "# 2. 生成一层全链接的神经网络。\n",
    "def nn_layer(input_tensor, input_dim, output_dim, layer_name, act=tf.nn.relu):\n",
    "    # 将同一层神经网络放在一个统一的命名空间下\n",
    "    with tf.name_scope(layer_name):\n",
    "        # 声明神经网络边上的权重，并调用生成权重监控信息日志的函数。\n",
    "        with tf.name_scope('weights'):\n",
    "            weights = tf.Variable(tf.truncated_normal([input_dim, output_dim], stddev=0.1))\n",
    "            variable_summaries(weights, layer_name + '/weights')\n",
    "            \n",
    "        # 声明神经网络的偏置项，并调用生成偏置项监控信息日志的函数。    \n",
    "        with tf.name_scope('biases'):\n",
    "            biases = tf.Variable(tf.constant(0.0, shape=[output_dim]))\n",
    "            variable_summaries(biases, layer_name + '/biases')\n",
    "            \n",
    "        with tf.name_scope('Wx_plus_b'):\n",
    "            preactivate = tf.matmul(input_tensor, weights) + biases\n",
    "            # 记录神经网络输出节点在经过激活的数之前的分布。\n",
    "            tf.summary.histogram(layer_name + '/pre_activations', preactivate)\n",
    "        activations = act(preactivate, name='activation')        \n",
    "        \n",
    "        # 记录神经网络输出节点在经过激活函数之后的分布。在图11.17中，对于layer1，因\n",
    "        # 为使用了ReLU函数作为激活函数，所以所有小于0的值部被设为了0。于是在激活后\n",
    "        # 的layer1/activations图上所有的值都是大于0的。而对于layer2，因为没有使用\n",
    "        # 激活函数，所以layer2/activations和layer2/pre_activations一样。\n",
    "        tf.summary.histogram(layer_name + '/activations', activations)\n",
    "        return activations\n",
    "    \n",
    "\n",
    "# 3. 主函数\n",
    "def main():\n",
    "    mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n",
    "    # 定义输入\n",
    "    with tf.name_scope('input'):\n",
    "        x = tf.placeholder(tf.float32, [None, 784], name='x-input')\n",
    "        y_ = tf.placeholder(tf.float32, [None, 10], name='y-input')\n",
    "\n",
    "    # 将输入向量还原成图片的像素矩阵，并通过tf.summary.image函数将当前的图片信息写入日志的操作\n",
    "    with tf.name_scope('input_reshape'):\n",
    "        image_shaped_input = tf.reshape(x, [-1, 28, 28, 1])\n",
    "        tf.summary.image('input', image_shaped_input, 10)\n",
    "\n",
    "    hidden1 = nn_layer(x, 784, 500, 'layer1')\n",
    "    y = nn_layer(hidden1, 500, 10, 'layer2', act=tf.identity)\n",
    "    \n",
    "    # 计算交叉熵并定义生成交叉熵监控日志的操作\n",
    "    with tf.name_scope('cross_entropy'):\n",
    "        cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=y, labels=y_))\n",
    "        tf.summary.scalar('cross_entropy', cross_entropy)\n",
    "\n",
    "    with tf.name_scope('train'):\n",
    "        train_step = tf.train.AdamOptimizer(0.001).minimize(cross_entropy)\n",
    "\n",
    "    # 当前模型在当前给定数据上的正确率，并定义生成正确率监控日志的操作。如果在sess.run\n",
    "    # 时给定的数据训练batch，那么得到的正确率就是在这个训练batch上的正确率；如果给定的\n",
    "    # 数据为验证或者测试数据，那么得到的正确率就是在当前模型在验证或者测试数据上的正确率。\n",
    "    with tf.name_scope('accuracy'):\n",
    "        with tf.name_scope('correct_prediction'):\n",
    "            correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "        with tf.name_scope('accuracy'):\n",
    "            accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "        tf.summary.scalar('accuracy', accuracy)\n",
    "\n",
    "    # 和TensorFlow其他操作类似，tf.summary.scalar、tf.summary.histogram和tf.summary.image\n",
    "    # 函数都不会立即执行，需要通过sess.run来明确调用这些函数。因为程序中定义的写日志操作\n",
    "    # 比较多，一一调用非常麻烦，所以TensorFlow提供了tf.summary.merge_all函数来整理所有的\n",
    "    # 日志生成操作。在TensorFlow程序执行的过程中只要运行这个操作就可以将代码中定义的所有\n",
    "    # 日志生成操作执行一次，从而将所有日志文件。\n",
    "    merged = tf.summary.merge_all()\n",
    "\n",
    "    with tf.Session() as sess:\n",
    "        # 初始化写日志的writer，并将当前TensorFlow计算图写入日志。\n",
    "        summary_writer = tf.summary.FileWriter(SUMMARY_DIR, sess.graph)\n",
    "        tf.global_variables_initializer().run()\n",
    "\n",
    "        for i in range(TRAIN_STEPS):\n",
    "            xs, ys = mnist.train.next_batch(BATCH_SIZE)\n",
    "            # 运行训练步骤以及所有的日志生成操作，得到这次运行的日志。\n",
    "            summary, _ = sess.run([merged, train_step], feed_dict={x: xs, y_: ys})\n",
    "            # 将得到的所有日志写入日志文件，这样TensorBoard程序就可以拿到这次运行所对应的\n",
    "            # 运行信息。\n",
    "            summary_writer.add_summary(summary, i)\n",
    "\n",
    "    summary_writer.close()\n",
    "    \n",
    "    \n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从以上程序可以看出，除了GRAPHS之外，Tensorboard中的每一栏对应了TensorFlow中一种日志生成函数，表11.1总结了这个对应关系：\n",
    "<p align='center'>\n",
    "    <img src=images/表11.1.JPG>\n",
    "</p>\n",
    "\n",
    "**1. SCALARS**\n",
    "\n",
    "运行以上样例程序并使用11.1节中介绍的方式启动TensorBoard，可以看到如下图所示的界面。在这个页面上展示了样例程序中通过`tf.summary.scalar`函数生成的所有标量监控信息。和变量的命名空间类似，TensorBoard也会根据监控指标的名称进行分组。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.13.JPG>\n",
    "    <center>图11-13 使用TensorBoard展示变量监控信息的默认页面</center>\n",
    "</p>\n",
    "\n",
    "图11.14中展示了将\"/.\\*/\"栏收起来之后的效果。可以看到，名称为layer1的栏目下有4组不同的监控指标。这4个不同的指标都以layer1开头，并通过斜线\"/\"划分不同的命名空间。不过和TensorFlow计算图可视化结果不同的是，SCALARS、IMAGES、AUDIO、TEXT、HISTOGRAMS和DISTRUBUTIONS栏只会对最高层的命名空间进行整合，单击展开后将看到该命名空间下的所有监控指标。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.14.JPG>\n",
    "    <center>图11-14 按命名空间整理的标量监控信息页面</center>\n",
    "</p>\n",
    "\n",
    "在每一个监控指标的左下角有一个小方框，单击这个方框可以得到放大后的图片。放大后的效果如下图11.15所示。再单击一次这个小方框可以将放大后的图表需缩小。在训练神经网络时，通过TensorBoard监控神经网络中变量取值的变化、模型在训练batch上的损失函数大小以及学习率的变化等信息可以更加方便地掌握模型的训练情况。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.15.JPG>\n",
    "    <center>图11-15 展开某一项监控标量时的放大图</center>\n",
    "</p>\n",
    "\n",
    "**2. IMAGES**\n",
    "\n",
    "图11.16展示了通过TensorBoard可视化当前轮训练使用的图像信息。通过这个界面可以大致看出数据随机打乱的效果。因为TensorFlow程序和TensorBoard可视化界面可以同时运行，所以从TensorBoard上可以实时看到TensorFlow程序中最新使用的训练或者测试图像。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.16.JPG>\n",
    "    <center>图11-16 通过TensorBoard可视化训练图像</center>\n",
    "</p>\n",
    "\n",
    "**3. DISTRIBUTIONS**\n",
    "\n",
    "TensorBoard的DISTRIBUTIONS一栏提供了对张量取值分布的可视化界面。通过图11.17这个界面，可以直观地观察到不同层神经网络中参数的取值变化。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.17.JPG>\n",
    "    <center>图11-17 通过TensorBoard可视化张量取值分布DISTRIBUTIONS效果图</center>\n",
    "</p>\n",
    "\n",
    "**4. HISTOGRAMS**\n",
    "\n",
    "为了更加清晰地展示参数取值分布和训练选代轮数之间的关系，TensorBoard提供了HISTOGRAMS视图。下图展示了HISTOGRAMS的效果图，图11.19展示了一个放大之后的参数取值分布和迭代轮数之间的关系。与DISTRIBUTIONS效果图不同，**HISTOGRAMS中不同轮数中参数的取值是通过不同的平面来表示的**。比如在图11.19中，颜色越深的平面表示迭代轮数越小的取值分布，比如图11.19中最上面的比较尖的平面表示训练一轮之后的bias参数取值分布。因为bias是通过全0矩阵初始化的，于是在第一轮时取值都集中在0附近。图11.19中最前面比较浅的平面表示迭代轮数较大时的参数取值分布。从图11.19中可以看到bias的取值分布越来越接近平均分布。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.18.JPG>\n",
    "    <center>图11-18 通过TensorBoard可视化张量取值分布HISTOGRAMS效果图</center>\n",
    "</p>\n",
    "<p align='center'>\n",
    "    <img src=images/图11.19.JPG>\n",
    "    <center>图11-19 HISTOGRAMS放大后的效果图</center>\n",
    "</p>\n",
    "\n",
    "在HISTOGRAMS视图左侧有一个“OVERLAY”选项，选择之后可以看到类似图11.20和图11.21所示的效果。和默认的OFFSET视图类似，在OVERLAY视图中颜色越深的表示迭代轮数越小。但是从图11.21中可以看到， 比较尖的曲线看上去颜色比较浅，而比较靠近平均分布的曲线反而比较深。这是因为有更多的曲线靠近平均分布，所以合在一起就比较深了。如图11.21中所示，当把鼠标移到某一条曲线上时这一条曲线就会变黑，而且迭代轮数的信息会显示在鼠标附近。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.20.JPG>\n",
    "    <center>图11-20 HISTOGRAMS视图OVERLAY模式效果图</center>\n",
    "</p>\n",
    "<p align='center'>\n",
    "    <img src=images/图11.21.JPG>\n",
    "    <center>图11-21 HISTOGRAMS视图OVERLAY模式放大后的效果图</center>\n",
    "</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
